Projecting patient image data from radioscopic imaging methods and/or tomographic imaging methods onto video images

ABSTRACT

The invention relates to a device for projecting patient image data from radioscopic imaging methods and/or tomographic imaging methods onto video images, comprising an image display device ( 10 ), at least one video camera ( 14 ) and a computer-assisted navigation system which can detect the spatial positions of the image display device ( 10 ) and/or the video camera ( 14 ) and the spatial positions of a part of a patient&#39;s body ( 1 ) via tracking means ( 12, 3 ) attached to it, wherein the image display device ( 10 ) and the video camera ( 14 ) are assigned to each other and are realized as a portable unit.

[0001] The present invention relates to a device for projecting patientimage data from radioscopic imaging methods and/or tomographic imagingmethods onto video images. The device comprises an image display device,at least one video camera and a computer-assisted navigation systemwhich can detect the spatial position of the image display device and/orthe video camera and the spatial positions of a part of a patient's bodyvia tracking means attached to it.

[0002] Such devices serve to visually assist a physician duringdiagnosis or while treating a patient. The option should be provided ofcombining images from the patient's interior with conventional videoimages, in order to facilitate treatment and diagnosis.

[0003] A video-based surgical target system is known from U.S. Pat. No.5,765,561, for which among other things the use of a tracked videocamera is proposed, whose images are superimposed onto assigned imagesfrom radioscopic imaging methods and/or tomographic imaging methods onthe statically arranged screen of a navigation system. A particulardisadvantage of such an embodiment is that the physician wishing to viewthe images always has to look away from the patient, to the screen ofthe navigation system. He can then no longer pay exact attention to theposition in which the camera is, i.e. the exterior position from whichhe is obtaining the superimposed images. The system is thereforesomewhat awkward and does not enable direct inspection within the partof the patient's body.

[0004] It is the object of the present invention to provide a device forprojecting patient image data from radioscopic imaging methods and/ortomographic imaging methods onto video images, which overcomes thedisadvantages of the prior art cited above. In particular, it should bemade possible for the treatment staff to obtain the desired image datadirectly and in direct view of the patient.

[0005] This object is solved in accordance with the invention by theimage display device and the video camera being assigned to each otherand realized as a portable unit. Using this embodiment in accordancewith the invention, the physician carrying out the treatment then hasthe option of holding the image display device, even with the assignedcamera, in front of the part of the patient's body, and therefore ofreading the desired anatomical information while looking directly at thepatient. It is possible in this respect to adjust the video image or theimage from the radioscopic imaging method and/or tomographic imagingmethod to be more or less transparent, as desired, in the image shown onthe image display device, in order to highlight specifically the desiredinformation most clearly within the display as a whole. Using the devicein accordance with the invention, another step is taken towards creatinga “glass patient”, simply by holding the image display device in frontof the relevant part of the patient's body. Since the camera assignmentto the image display device is tracked via the tracking means, theposition of these two elements is always known, and the navigationsystem has information on how and from which direction the part of thepatient's body is being viewed. In particular when performing minimallyinvasive surgery in which the area of surgery is not exposed, the devicein accordance with the invention thus offers the physician carrying outthe treatment the option of again informing himself of the exactposition of the parts of the patient's anatomy during the operation. Inthis way, tracked or navigated instruments can of course also been shownon the image display device. The physician no longer has to look awayfrom the patient in order to look into the interior of the patient'sbody.

[0006] In accordance with a preferred embodiment of the invention, thevideo camera is arranged on the rear side of the image display device.This results in a compact apparatus; the camera can be completelyremoved from the operator's field of view. It should be noted inprinciple that the camera can be attached at any point on the rear sideof the image display device, or can be integrated into it, for exampleeven in the area of the edge of the image display device.

[0007] The video camera is preferably spatially assigned to the imagedisplay device in a way which is predetermined and/or known to thesystem, and the tracking means is arranged on the image display device.Spatially assigning the video camera and the image display device in away which is predetermined and/or known to the system opens up theoption of tracking just one of these two apparatus by means of thetracking means, since it is also known exactly where the other apparatusis situated. The designer is then also at liberty to arrange thetracking means either on the image display device itself or on the videocamera, according to how said tracking means can best be detected by thenavigation system. In optically based navigation systems, the trackingmeans can also be an arrangement of reflecting or actively emittingmarkers, although it is of course also possible within the context ofthe present invention to perform navigation via a magnetic trackingsystem in which coils are tracked in a generated magnetic field.

[0008] In order to design the image display device to be easy to handle,it can be designed as a portable LCD flat screen.

[0009] Data transmission between the individual apparatus of the devicein accordance with the invention is realized in different ways. On theone hand, it is possible to communicate the image data from theradioscopic imaging method and/or tomographic imaging method to beprojected onto the video images to the image display device via radiointerfaces of the navigation system, while on the other handcommunicating the data via data cables is also conceivable. Oneembodiment ought to prove particularly advantageous within the contextof handling the device in accordance with the invention, in which theimage display device is provided with its own energy supply (battery orpower pack) and transmission is by radio, since this achieves the mostextensive freedom of handling.

[0010] In accordance with a preferred embodiment, the video camerashould exhibit a small aperture and a low depth of field, so that only asmall area of the detected image is in the focal plane. Using such acamera, it is possible to determine the distance between the image andthe focal plane. If the spatial position of the camera is known from thenavigation system, and the distance from the image plane is also known,then the computer system connected to the navigation system cancalculate the spatial position of the video image in real time. Thisenables the video image and the image data from a radioscopic imagingmethod and/or tomographic imaging method carried out beforehand to beoptimally assigned.

[0011] In accordance with another advantageous embodiment of the devicein accordance with the invention, a data transmission means is arrangedon the video camera or on the image display device, wherein informationon the image shown on the image display device is transmitted to thenavigation system by means of said data transmission means. The imagedata to be projected, communicated from the navigation system to theimage display device, are then adapted in size and position on the imagedisplay device such that they are in registration with the video images,i.e. a 1:1 representation of the two images in overlap is created. Tothis end, a contour-matching unit can be provided in the navigationsystem, said contour-matching unit superimposing the video images andthe projected images, in particular via outer contour matching of thepart of the patient's body.

[0012] It should also be noted here that it may be conceivable andadvantageous within the context of the invention to attach to theportable unit many and various control apparatus and command inputapparatus for altering the combined image shown. For example, the focalplane both of the radioscopic imaging method or tomographic imagingmethod employed and of the video image shown can be altered using suchcontrol elements. As already mentioned above, it is also possible toprovide control elements which alter the respective degree oftransparency of one of the images with respect to the other.

[0013] An illuminating device for the part of the patient's body isadvantageously provided on the camera or in its vicinity on the imagedisplay device, especially for example as a ring light (LEDs, lamps,fluorescent tubes).

[0014] The invention will now be described in more detail, by way of anembodiment. In the drawings provided for this purpose, there is shown:

[0015]FIG. 1 a schematic representation of the use of a device inaccordance with the invention for viewing the knee joint of a patient,in aspect;

[0016]FIG. 2 a lateral view of the device according to FIG. 1, togetherwith a schematically represented navigation system; and

[0017]FIG. 3 a flow diagram of operating the device in accordance withthe invention.

[0018]FIG. 1 schematically shows a device in accordance with theinvention, using which the interior of a patient's joint may for examplebe inspected. The leg 1 of the patient is to be examined at the joint 5.To this end, the flat screen 10 designed in accordance with theinvention is held directly in front of the joint, and an image thenappears on the screen 10 containing both image information from thevideo camera image of the camera 14 (FIG. 2) as well as imageinformation on the interior anatomical structure, said information beingcommunicated for example by a navigation apparatus, by radio. To thisend, a tomographic image (MR or CT image) is produced beforehand of thepatient or of the patient's leg 1.

[0019] As is indicated more clearly in FIG. 2, the camera 14 issituated, firmly attached, on the rear side of the screen 10.

[0020] This type of image representation is made possible by the spatialposition of the camera 14 or of the image screen 10 being tracked aroundthe leg 1 of the patient by means of a tracking means. In the presentcase, this tracking means is an optical tracking means, namely anarrangement 12 of markers whose position is detected by means of theschematically represented navigation system 20. To this end, thenavigation system 20 includes for example two spaced-out infraredcameras 24, 26 and an infrared light emitter 22. The three-dimensionalspatial position of the arrangement 12 of markers, and therefore also ofthe screen 10 and the firmly installed camera 14, can be determined fromthe two images from the infrared cameras 24, 26. Furthermore, it ispossible to determine the spatial position of the leg 1 using acorresponding arrangement 3 of markers.

[0021] If the position of the screen 10 and the camera 14 is then known,the position of the current image plane can be detected as follows. Thecamera 14 has, namely, a small aperture and a low depth of field, suchthat the image only appears in focus in a particular focal plane. Thisallows the position of said image plane to be determined and an imagefrom the CT or MR imaging method to be transmitted on the screen 10,which is also exactly in said image plane. The important thing here isthat the image ratio of the video image is shown at a scale of 1:1 withrespect to the tomographic image. If the joint of a patient is thenviewed using the device in accordance with the invention, one sees forexample the contours of the patient's leg at the upper and lower edge ofthe screen. Using this information, the image from the tomographicimaging method can then be adapted such that it lies exactly over thevideo image, to ensure a realistic impression. The person viewing thescreen 10 can then view on the screen both the video image, i.e. a realimage, and the “interior” image from the tomographic imaging method, andin any desired depth and with any adjusted transparency of the twoimages. These adjustments can be made via control devices (not shown),for example flip switches on the front side of the screen.

[0022] For the first time, therefore, the device in accordance with theinvention gives the observer the option of looking into the interiortreatment area in real time while looking at the patient, and ofplanning or adapting the treatment accordingly. He no longer has to lookaway from the patient to a firmly installed navigation screen, which isparticularly advantageous when the patient's responses to particularmedical measures are to be tested. Since the “interior” images from theradioscopic imaging method and/or tomographic imaging method are formedsuch that they represent virtual 3-D views of the interior parts of thebody, the image as a whole can be made very vivid.

[0023] Although a radioscopic and/or tomographic image is therefore nottaken in situ, a virtual “glass patient” is thus created.

[0024]FIG. 3 shows the sequence of a method such as is performed usingthe device in accordance with the invention; the activity carried out bythe computer of the navigation system is illuminated here in particular.The upper left side of the flow diagram relates to the processes invideo image detection, while on the right-hand side image detection forthe interior of the body is explained, using the example of MR/CT imagedetection. In video image detection, the focal plane is firstlydetermined, as has already been explained above. Then the distance fromthe focal plane to the camera is determined, whereupon the camera isregistered by means of its tracking means in the navigation system. Thedetermined spatial position of the camera with respect to the navigationsystem is then known.

[0025] With respect to MR/CT image detection, an MR/CT image is firstlydetected beforehand, whereupon a three-dimensional image isreconstructed from the data obtained. Then, the object—for example, apart of the patient's body—is spatially registered, i.e. the spatialrelation of the position of the object to the MR/CT images isdetermined. This can only be realized by spatially detecting anarrangement of markings situated on the object.

[0026] When the video image detection and MR/CT image processing stepscited above have been performed, then video/MR/CT-image matching isperformed, i.e. the two images are harmonized under computer guidance,with respect to image position and image size. Then, the MR/CT/3-Dimages, projected onto the video images, can be superimposed on theimage display device, and can assist the physician performing thetreatment in diagnose and treatment.

1. A device for projecting patient image data from radioscopic imagingmethods and/or tomographic imaging methods onto video images, comprisingan image display device (10), at least one video camera (14) and acomputer-assisted navigation system which can detect the spatialpositions of said image display device (10) and/or said video camera(14) and the spatial positions of a part of a patient's body (1) viatracking means (12, 3) attached to it, characterized in that said imagedisplay device (10) and said video camera (14) are assigned to eachother and are realized as a portable unit.
 2. The device as set forth inclaim 1, characterized in that said video camera (14) is arranged on therear side of said image display device (10).
 3. The device as set forthin any one of claims 1 or 2, characterized in that said video camera(14) is spatially assigned to said image display device (10) in a waywhich is predetermined and/or known to the system, and in that saidtracking means (12) is arranged on said image display device (10). 4.The device as set forth in any one of claims 1 or 2, characterized inthat said video camera (14) is spatially assigned to said image displaydevice (10) in a way which is predetermined and/or known to the system,and in that said tracking means (12) is arranged on said video camera(14).
 5. The device as set forth in any one of claims 1 to 4,characterized in that said image display device (10) is a portable LCDflat screen.
 6. The device as set forth in any one of claims 1 to 5,characterized in that the image data from said radioscopic imagingmethod and/or tomographic imaging method to be projected onto said videoimages are communicated to said image display device (10) via radiointerfaces of said navigation system.
 7. The device as set forth in anyone of claims 1 to 5, characterized in that the image data from saidradioscopic imaging method and/or tomographic imaging method to beprojected onto said video images are communicated to said image displaydevice (10) via a data cable of said navigation system.
 8. The device asset forth in any one of claims 1 to 7, characterized in that said videocamera (14) exhibits a small aperture and a low depth of field.
 9. Thedevice as set forth in any one of claims 1 to 8, characterized in that adata transmission means is arranged on said video camera (14) or on saidimage display device (10), by means of which information on the imageshown on said image display device (10) is transmitted to saidnavigation system, wherein the image data to be projected, from saidradioscopic imaging methods and/or tomographic imaging methods,communicated from said navigation system to said image display device(10), are adapted in size and position on said image display device (10)such that they are in registration with said video images.
 10. Thedevice as set forth in claim 9, characterized in that said navigationsystem comprises a contour-matching unit which superimposes said videoimages and said projected images, in particular via outer contourmatching of said part of the patient's body.
 11. The device as set forthin any one of claims 1 to 10, characterized in that an illuminatingdevice for said part (1) of the patient's body is provided on saidcamera (14) or in its vicinity on said image display device (10); inparticular, a ring light consisting of LEDs, lamps or fluorescent tubes.